Using ChatGPT to evaluate cancer myths and misconceptions: artificial intelligence and cancer information

Abstract Data about the quality of cancer information that chatbots and other artificial intelligence systems provide are limited. Here, we evaluate the accuracy of cancer information on ChatGPT compared with the National Cancer Institute’s (NCI’s) answers by using the questions on the “Common Cancer Myths and Misconceptions” web page. The NCI’s answers and ChatGPT answers to each question were blinded, and then evaluated for accuracy (accurate: yes vs no). Ratings were evaluated independently for each question, and then compared between the blinded NCI and ChatGPT answers. Additionally, word count and Flesch-Kincaid readability grade level for each individual response were evaluated. Following expert review, the percentage of overall agreement for accuracy was 100% for NCI answers and 96.9% for ChatGPT outputs for questions 1 through 13 (ĸ = ‒0.03, standard error = 0.08). There were few noticeable differences in the number of words or the readability of the answers from NCI or ChatGPT. Overall, the results suggest that ChatGPT provides accurate information about common cancer myths and misconceptions.

Patients are increasingly turning to the internet for information about cancer, with 80% of US adults reportedly using the internet to seek health information (1). Within the online communication environment, cancer misinformation and harmful information remain a serious concern (2). Chatbots and other artificial intelligence (AI) systems have become increasingly popular in recent years for providing information and assistance to users in online spaces (3); recently, 1 such AI system receiving a lot of public and news attention is ChatGPT, which uses natural language processing to generate responses to user input. Given the importance of accurate information in the field of cancer research and treatment, determining the accuracy of AI (mis)information outputs from chat platforms such as ChatGPT is critical to clinicians and, more broadly, health and medical communicators. As an initial step in monitoring these public platforms, we engaged in a smallscale study of the information ChatGPT generated.
Using the National Cancer Institute's (NCI's) web page "Common Cancer Myths and Misconceptions" (4), on December 20, 2022, we asked ChatGPT (December 15, 2022, version) 13 questions about cancer that are common points of confusion among the public (per NCI). The NCI's answers and ChatGPT answers to each question were blinded, and then evaluated for accuracy based on the established knowledge of 5 scientific reviewers with expertise in cancer treatment and cancer misinformation (B.K., C.B., A.K., E.W., and S.A.) between December 20, 2022, and January 3, 2023. Expert reviewers were not compensated. Accuracy ratings (yes vs no) were evaluated independently for each question, and then compared between the blinded NCI and ChatGPT answers. Interrater agreement was evaluated by Fleiss OE (5). Statistical analyses were performed using Stata, version 17 (StataCorp LP).
Following expert review, 13 of 13 NCI answers were rated as accurate by 5 of 5 expert reviewers, demonstrating 100% interrater agreement. ChatGPT outputs were determined to be accurate by 5 of 5 reviewers for 11 of 13 questions, demonstrating 96.9% overall agreement. The remaining 2 questions (question 5, "Does my attitude-positive or negative-determine my risk of getting or surviving cancer?" and question 9, "Do power lines cause cancer?") were rated as accurate by 4 of 5 reviewers, with a single unique reviewer rating each question as inaccurate (OE ¼ -0.03, SE ¼ 0.08, P ¼ .6). Table 1 displays all questions and associated answers from NCI and ChatGPT, respectively, as well There is no scientific evidence to support the idea that eating sugar can directly cause cancer or make it worse. However, there is some evidence to suggest that high sugar intake may be associated with an increased risk of certain types of cancer, such as breast and colon cancer. This is because a diet high in sugar is often associated with being overweight or obese, and carrying excess weight has been linked to an increased risk of cancer. Additionally, high sugar intake may be linked to inflammation in the body, which has also been associated with an increased risk of cancer. However, it is important to note that the link between sugar intake and cancer risk is still being studied, and more research is needed to understand the relationship between these factors. It is generally recommended to limit your intake of added sugars, such as those found in sugary drinks, sweets, and baked goods, as part of a healthy diet. This is because a diet high in added sugars has been linked to an increased risk of several health problems, including tooth decay, type 2 diabetes, and heart disease. Instead of sugary foods, it is important to focus on eating a diet that is rich in a variety of vegetables, fruits, whole grains, and lean protein sources. This can help you maintain a healthy weight and reduce your risk of chronic diseases, including cancer. 234 13.7 Do artificial sweeteners cause cancer?
No. Researchers have conducted studies on the safety of the artificial sweeteners (sugar substitutes) saccharin (Sweet 'N Low V R , Sweet Twin V R , NectaSweet V R ); cyclamate; aspartame (Equal V R , NutraSweet V R ); acesulfame potassium (Sunett V R , Sweet One V R ); sucralose (Splenda V R ); and neotame and found no evidence that they cause cancer in humans. All of 80 9.5 There is no clear evidence that artificial sweeteners cause cancer. Some studies have suggested a link between the use of artificial sweeteners and an increased risk of certain types of cancer, while other studies have not found such a link. The results of these studies have been mixed, and more research is needed to understand the relationship between artificial sweeteners and cancer. The U.S. Food and Drug Administration (FDA) has approved several artificial sweeteners as safe for consumption, including aspartame, saccharin, and sucralose. These sweeteners have been extensively studied and have been found to be safe for human consumption at the levels typically consumed. However, it is important to note that some people may be sensitive to certain artificial sweeteners and may experience side effects such as headaches, digestive issues, or allergic reactions. If you are concerned about the safety of artificial sweeteners, it is always a good idea to talk to your healthcare provider for guidance.

Is cancer contagious?
In general, no. Cancer is not a contagious disease that easily spreads from person to person. The only situation in which cancer can spread from one person to another is in the case of organ or tissue transplantation. A person who receives an organ or tissue from a donor who had cancer in the past may be at increased risk of developing a transplantrelated cancer in the future. However, that risk is extremely low-about two cases of cancer per 10,000 organ transplants. Doctors avoid the use of organs or tissue from donors who have a history of cancer. In some people, cancers may be caused by certain viruses (some types of human papillomavirus, or HPV, for example) and bacteria (such as Helicobacter pylori The chance that surgery will cause cancer to spread to other parts of the body is extremely low. Following standard procedures, surgeons use special methods and take many steps to prevent cancer cells from spreading during biopsies or surgery to remove tumors. For example, if they must remove tissue from more than one area of the body, they use different surgical tools for each area. For information about how cancer spreads in the body, see our page on Metastatic Cancer. 80 11.0 In most cases, cancer surgery or a tumor biopsy will not cause cancer to spread in the body. In fact, these procedures are typically done to remove or diagnose cancer that is already present in the body. During surgery, the goal is to remove as much of the cancer as possible, including any visible tumors and any cancer cells that may have spread to nearby tissues. A biopsy is a procedure in which a small sample of tissue is removed and examined under a microscope to determine if cancer is present. In some cases, it is possible that cancer cells may be released into the body during surgery or a biopsy. This can occur when cancer cells are present in the fluids or tissues surrounding the tumor. However, it is important to note that these cells are usually present in very low numbers and are not likely to cause the cancer to spread. It is important to work with a healthcare team to determine the best treatment approach for your situation. This may include surgery, chemotherapy, radiation, or other treatments, depending on the type and stage of your cancer. Your healthcare team can provide more information about the potential risks and benefits of these treatments and help you make the best decision for your situation.

13.4
Will cancer get worse if exposed to air?
No. Exposure to air will not make tumors grow faster or cause cancer to spread to other parts of the body. For information about how cancer spreads in the body, see our page on Metastatic Cancer. 36 8.8 Exposing cancer to air will not cause it to get worse. Cancer is a disease in which abnormal cells grow and divide uncontrollably, forming tumors. These abnormal cells are present in the body, and they do not require air to grow or spread. It is important to note that cancer surgery may involve exposing cancerous tumors to air. During surgery, the goal is to remove as much of the cancer as possible, including any visible tumors and any cancer cells that may have spread to nearby tissues. In some cases, the cancer may be removed by cutting into the skin, which can expose the cancerous tissue to air. In other cases, the cancer may be removed through laparoscopic surgery, which involves making small incisions in the skin and inserting instruments and a camera through the incisions.  There is no clear evidence that cell phones or other wireless devices cause cancer. Some studies have suggested a possible link between the use of cell phones and an increased risk of certain types of cancer, such as brain cancer. However, other studies have not found such a link. The World Health Organization (WHO) has classified radiofrequency (RF) fields, which are emitted by cell phones and other wireless devices, as possibly carcinogenic to humans. However, this classification is based on limited evidence from studies of high RF field exposure in animals, and it does not mean that these fields definitely cause cancer. The WHO has also noted that the levels of RF fields emitted by cell phones are typically much lower than those that have been associated with increased cancer risk in animal studies. More research is needed to understand the potential health effects of RF fields and the use of cell phones.
In the meantime, it is generally recommended to take steps to reduce your RF field exposure, such as using hands-free devices or texting instead of talking on the phone, and keeping your phone away from your body when it is not in use.

15.3
Do power lines cause cancer?
No, not according to the best studies completed so far. Power lines emit both electric and magnetic energy. The electric energy emitted by power lines is easily shielded or weakened by walls and other objects. The magnetic energy emitted by power lines is a low-frequency form of radiation that does not damage genes. For more information, see the NCI fact sheet on Electromagnetic Fields and Cancer.

13.2
There is no clear evidence that living near power lines causes cancer. Some studies have suggested a possible link between living near power lines and an increased risk of certain types of cancer, such as leukemia and brain cancer. However, other studies have not found such a link. Power lines emit extremely low frequency (ELF) electromagnetic fields (EMFs), which are a type of nonionizing radiation. Non-ionizing radiation is generally considered to be less harmful than ionizing radiation, which is known to cause DNA damage and increase the risk of cancer. The levels of ELF EMFs emitted by power lines are typically much lower than the levels that have been associated with increased cancer risk in animal studies. More research is needed to understand the potential health effects of ELF EMFs and the risk of cancer. In the meantime, it is generally recommended to take steps to reduce your exposure to all types of EMFs,  Not necessarily. Cancer is caused by harmful changes (mutations) in genes. Only about 5 to 10% of cancers are caused by harmful mutations that are inherited from a person's parents. In families with an inherited cancer-causing mutation, multiple family members will often develop the same type of cancer. These cancers are called "familial" or "hereditary" cancers. The remaining 90 to 95% of cancers are caused by mutations that happen during a person's lifetime as a natural result of aging and exposure to environmental factors, such as tobacco smoke and radiation. These cancers are called "non-hereditary" or "spontaneous" cancers. For more information about the risk of getting cancer, see the NCI fact sheet on Genetic Testing for Inherited Cancer Susceptibility Syndromes and Cancer Causes and Risk Factors.
Having a family member with cancer may increase your risk of developing cancer, but it does not guarantee that you will develop the disease. The risk of developing cancer can be influenced by a variety of factors, including genetics, environmental exposures, and lifestyle habits. If you have a family member with cancer, it is important to be aware of your personal risk factors and to take steps to reduce your risk of cancer. This may include adopting healthy habits such as not smoking, maintaining a healthy weight, and exercising regularly. It may also be helpful to speak with your healthcare provider about your family medical history and any recommended screening tests or preventive measures. It is also important to remember that cancer is a complex disease, and not all cases are caused by genetics. Many people who develop cancer do not have a family history of the disease, and conversely, some people who have a family history of cancer may never develop the disease.

17.4
If no one in my family has had cancer, does that mean I'm risk-free?
No. Based on the most recent data, about 38% of men and women will be diagnosed with cancer at some point during their lives. Most cancers are caused by genetic changes that occur throughout a person's lifetime as a natural result of aging and exposure to environmental factors, such as tobacco smoke 90 11.8 Having a family history of cancer does not guarantee that you will develop the disease, and not having a family history does not mean that you are risk-free. While genetics can play a role in the development of cancer, there are many other factors that can influence your risk of the disease, including There is no clear evidence that the use of antiperspirants or deodorants causes breast cancer. Some studies have suggested a possible link between the use of these products and an increased risk of breast cancer, but the results of these studies have been mixed and more research is needed to understand the relationship between antiperspirants and deodorants and breast cancer. The National Cancer Institute (NCI) states that there is currently no strong evidence to support a link between the use of antiperspirants or deodorants and breast cancer. The NCI notes that while some studies have suggested a possible link between the use of these products and an increased risk of breast cancer, the results of these studies have been inconclusive and more research is needed to understand the relationship between these products and breast cancer. If you are concerned about the safety of antiperspirants or deodorants, it is always a good idea to speak with your healthcare provider for guidance. Your healthcare provider can provide more information about the potential risks and benefits of these products and can help you make an informed decision about your personal care. as word count and Flesch-Kincaid readability grade level for each individual response. There were few noticeable differences in the number of words or the readability of the answers from NCI or ChatGPT. Answers from both sources were written at a less readable rate than would likely be recommended by health literacy advocates, for example. Although our team did not code for specific message features, multiple observers noted their perception that the ChatGPT messages often used terms associated with hedging and uncertainty that could lead to the adoption of a belief or practice that could be harmful. Overall, the results suggest that output from ChatGPT provides accurate information about common cancer myths and misconceptions without providing misinformation or harmful information to users. To evaluate whether repeated questioning could result in outputs with semantic variations that could lead to misinformation, we completed 5 runs repeatedly from the same input for each of the 13 questions (Supplementary Table 1, available online). On review, although subtle sentence structure changes were noted, the answers remained consistent, suggesting that answers remain accurate with repetitive questioning. This brief report offers important insights into the potentially positive capabilities of ChatGPT and other AI systems in the context of cancer-related (mis)information. As a team, we were initially uncertain about the utility of ChatGPT for health information and thought that the AI system could unintentionally convey misinformation or harmful information to users. This possibility is still important to consider for any AI chat system, given that users can problematically shape chatbot responses in a short period (6); researchers have found that AI systems can "enact malignant stereotypes" (7); and, specific to ChatGPT, there is reasonable skepticism about the tool's utility (8) when considering a breadth of topics beyond the basic cancer information we asked about in our study. ChatGPT could be the future of AI chat technology (or perhaps the tool is just a popular example of the state of the technology's capabilities); regardless of this particular tool's features, however, we must consider how best to monitor and evaluate the use of these tools in the online communication environment. Future systematic work to evaluate use and outputs of such platforms requires an infrastructure to monitor these sites, their algorithms and information sources, and potential bias in providing information equitably to diverse populations. There is concern that algorithms reinforce current health disparities and inequities (9), although the extent to which this is true about health information specifically is currently unknown. Other practical considerations about how people will interpret AI-generated chat responses in terms of trust and credibility (10) exist, particularly as they relate to medical and health information, behaviors, and services compared with consumer products, behaviors, and services.
The limitations of this study are that we evaluated only common cancer misinformation in the English language. More obscure cancer myths may have incomplete or English-only information, resulting in the model's inability to be trained on a sufficiently large and diverse data set to answer fewer common questions or non-English-language questions accurately. Finally, although this study obtained contemporary ChatGPT outputs, there is a possibility that these outputs were trained on models that could be scientifically outdated. Currently, ChatGPT is limited to data collected before 2021, so it is possible that as new scientific information emerges, ChatGPT may not be an accurate source of information or, at the least, its accuracy will be delayed.
Collectively, we found ChatGPT outputs in response to common cancer misinformation to be accurate and similar to answers that the NCI provided. In the context of these questions, there does not appear to be a clear area where this system may be susceptible to misinformation. Future research is needed to determine whether other chatbots and AI-driven systems provide accurate cancer information consistently, whether these findings apply to more diverse claims about cancer across the continuum, and what the ideal infrastructure might be for future monitoring to ensure accuracy of cancer information within the online information ecosystem.

Data availability
The data underlying this article are available online, and the data sets were derived from sources in the public domain: https:// chat.openai.com and https://www.cancer.gov/about-cancer/ causes-prevention/risk/myths.

Funding
This study was funded, in part, by the Huntsman Cancer Institute.

Conflicts of interest
All authors had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Skyler B. Johnson, MD, has no financial conflicts of interest. All other researchers reported no financial conflicts of interests.